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Abstract 

Planning, conducting, and reporting leading-edge research requires professionals who are 
capable of highly skilled reading. This study reports the development of an empirically infonned 
computer-based learning environment designed to foster the acquisition of reading 
comprehension strategies that mediate expertise in the social sciences. Empirical data were 
gathered in a mixed-methods explanatory sequential design that examined the reading 
comprehension strategies used by an expert social scientist while reading a professional-level 
text. Process data were collected through a concurrent think-aloud protocol and coded according 
to reading comprehension processes. We combined both quantitative and qualitative analyses to 
identify, describe, and explain patterns in the expert’s use of reading strategies. Our findings 
indicate that highly-skilled reading is characterized by critiquing text information, relating 
information to prior knowledge, and evaluating one’s own understanding of text information. 
Findings are used to inform the design of worked-examples and a pedagogical agent embedded 
within the Highly-Skilled Reading Tutor. 
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Resume 

Le type de planification, de realisation et Tanalyse qui caracterise une recherche d’avant-garde 
necessite des professionnels en mesure d’effectuer des lectures hautement specialisees. La 
presente etude dresse un rapport sur Elaboration d’un milieu d’apprentissage informatise concu 
pour favoriser Tacquisition de strategies de comprehension en lecture pennettant d’assurer la 
transmission des connaissances specialisees en sciences sociales. La collecte de donnees 
empiriques s’est effectuee suivant une conception sequentielle explicative fondee sur une 
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methode mixte, qui etudiait les strategies de comprehension de lecture utilisees par un expert en 
sciences sociales lors de sa lecture d’un texte de calibre professionnel. La collecte des donnees 
sur le processus s’est effectuee suivant un protocole concurrent de reflexion a haute voix, et les 
donnees ont ete codees conformement aux processus de comprehension de la lecture. Nous avons 
combine les analyses quantitatives et qualitatives a fin d’identifier, decrire et expliquer les 
tendances de cet expert dans l’utilisation des strategies de lecture. Nos resultats indiquent que la 
lecture hautement specialisee se caracterise par la critique des infonnations presentees dans le 
texte, la mise en relation des informations presentees et des connaissances anterieures et 
1’autoevaluation de la comprehension de ces informations. Les resultats obtenus sont utilises 
pour fonnuler des exemples faconncs et creer un agent pedagogique integre au Tuteur de lecture 
hautement specialisee. 


Introduction 

Advances in research and development depend on the training of professionals and academics 
that are capable of planning, reporting, and conducting innovative research. As such, post¬ 
secondary education plays a crucial role, in light of the current global economic downturn, 
because it ensures that Canadians have the knowledge and skills that is required to maintain a 
sustainable economy (Canadian Council on Learning, 2009). However, policy and research 
organizations in North America, across both public and private sectors, have highlighted a 
pressing need to improve the graduate training of educational researchers for more than a decade 
(e.g., Carnegie Foundation, 2003; Institute for Education Sciences, 2004; National Research 
Council, 2002, 2004). 

Amongst the many skills that undergraduate students are expected to acquire during their studies, 
the ability to analyze, synthesize, and critique scientific articles is especially critical. The existing 
research shows that expert readers use knowledge from their own disciplines to guide their 
reading - knowledge of how to create, represent, and evaluate scientific knowledge (Shanahan, 
2009; Shanahan & Shanahan, 2008). In the social sciences and humanities, professional-level 
scientific articles follow a typical structure, which consists of an introduction, methods, results, 
and discussion. In these manuscripts, social scientists develop and evaluate research hypotheses 
and draw conclusions based on their findings. While reading such manuscripts, social scientists 
use and adjust a wide range of strategies, such as summarizing, elaborating, and seeking 
clarifications (for a review, see Wyatt et ah, 1993) and differ from other academic disciples 
(Voss, Wiley, & Carretero, 1995). Their reading is responsive in that it is dependent on their 
reading goal, prior knowledge, and reactions to the content of the manuscript. 

Although important advances have been made in terms of the identification, classification, and 
description of strategies that mediate expertise in reading scientific articles in the social sciences, 
little is known in regards to how novices are able to acquire the strategies that are used by 
experts (Pressley & Lundeberg, 2008). Since the seminal Wyatt et al. (1993) study first 
investigated the reading strategies used by social scientists, researchers have yet to translate their 
findings to guide the design of reading interventions. We believe that the lack of such an 
educational intervention is critical given that the ability to engage in highly skilled reading is a 
crucial skill for researchers to advance scientific knowledge in their respective fields. 
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In this paper, we begin to address this issue by using advanced learning technologies as a 
platfonn to train education researchers that are capable of highly skilled reading. In doing so, this 
research addresses the following research questions: (1) how can we detect, capture, and 
represent disciplinary comprehension in the social sciences and humanities? (2) how can we 
assist students in studying and acquiring examples of disciplinary comprehension with advanced 
learning technologies? 

In order to answer these questions, our research has two objectives: our first objective is to 
outline a theory-driven approach to capture the reading comprehension strategies that mediates 
expertise in the social sciences and humanities based on the methodology developed by Wyatt et 
al. (1993); and our second objective is to use example-based skill acquisition as an instructional 
framework to derive empirically-based design guidelines in developing a computer-based 
learning environment called the Highly-Skilled Reading Tutor (Hi-Ski Tutor). The software aims 
to assist novices to acquire the strategies used by experts while reading professional-level texts in 
the social sciences and humanities. 

Method 

Research Design 

Single-subject mixed methods case studies and experimental research designs enable literacy 
researchers to translate their findings to practice by exhaustively capturing the complexity of the 
phenomenon under study in an ecologically valid setting (Neuman & McCormick, 2002). This 
research study follows a mixed-methods explanatory sequential design (N= 1) in order to 
investigate the reading comprehension strategies used by a highly skilled social scientist while 
reading a scientific article. The deployment of macro-level reading comprehension strategies is 
analyzed quantitatively in terms of frequency across time (i.e., transient analysis of reading 
strategies based on a 5 minute interval) and probabilities of shifts in strategies (i.e., one- and 
two-state transitions). Based on the findings of the quantitative analysis, the qualitative analysis 
aims to describe the micro-level reading comprehension strategies to better capture the reading 
path at finer levels of granularity. 

Participant 

A tenured full social science professor from a research-intensive public Canadian university was 
selected based on an exemplary level of proficiency in his/her own field of research. Following 
Wyatt et al. (1993), the identification of a pool of potential expert readers was accomplished 
according to two criteria: (a) possessing a doctorate in a social or behavioral science, and (b) 
publishing at least five articles in selective outlets over the last five years. The professor met the 
aforementioned criteria; furthennore, he/she was recognized internationally for high-ranking 
scholarly productivity in Contemporary Educational Psychology (Jones et al., 2010) and has 
extensive editorial experience. The area of specialization of the expert did not focus on reading 
strategies and skilled reading. The participant was selected using convenience sampling, which 
involved finding and meeting him/her through the faculty listing posted on the departmental 
website. 
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Material 

The scientific article read by the participant followed a classic or typical social science structure 
(for review, see Pressley & Lundeberg, 2008) and was published in a peer-reviewed journal. The 
article mentioned a title as well as the author(s) and included an abstract, introduction, methods, 
results, discussion, and references section. First, the abstract summarized the main findings of 
the study as well as details surrounding the designs and methods that were utilized. The 
introduction then situated the study within the broader scientific literature regarding the 
phenomena under investigation. The methods section explained what the participants underwent 
as part of the experiment. Next, the results section summarized the main findings as well as the 
statistical analyses that were performed. The discussion section li nk ed these main findings to 
previous findings obtained in the scientific literature and outlined their significance as well as 
their limitations. At the end of the article, the references section provided references to all the 
citations included as part of the article. 

Measure 

In order to measure the deployment and fluctuations in the use of reading comprehension 
strategies while reading professional-level texts, we elicited, recorded, and transcribed 
concurrent think aloud protocols (Afflerbach & Pressley, 1995; Ericsson & Simon, 1980, 1993). 
Concurrent thi nk aloud protocols involve asking participants to thi nk aloud or verbalize their 
own thought processes while they are reading the scientific article. The data obtained from these 
verbal reports can be both analyzed quantitatively by segmenting and categorizing a protocol 
using a coding scheme and qualitatively by depicting verbal reports in the form of a visual 
representation (Chi, 1997). The think-aloud procedure is an extensively validated research tool 
for reading comprehension processes (Afflerbach, 2002; Cote & Goldman, 1999; Magliano & 
Graesser, 1991; Magliano & Millis, 2003; Magliano, Trabasso, & Graesser, 1999; Zwaan & 
Brown, 1996), and when proper protocol is followed, is non-intrusive on natural cognition (for 
an extensive review, see Ericsson, 2006; Ericsson & Simon, 1993). 

Procedure 

Following Wyatt et al.’s (1993) study, the participant was informed about the aim of the study 
(i.e., "To investigate how experts stay current in their fields of expertise"). The participant was 
asked to select three research articles that were not read beforehand but would be of professional 
interest to read to stay current within his/her relevant field. The participant was also instructed to 
not begin reading the article; rather only choose each based on the title and author(s) only. There 
was no time restriction to find these articles. Upon identifying three articles, the participant 
infonned the researchers which articles he/she had chosen and the experimenter verified to make 
sure that the articles had an empirical research component and were published in a peer-reviewed 
scholarly journal. This was indeed the case for all three articles chosen by the professor and two 
copies of each article were made available for the data collection session. 

At the beginning of the experiment, the experimenter asked the participant to choose which one 
of the three articles he/she wanted to read. The participant was informed that if the article was 
not interesting, he/she could stop reading and select another one. However, this did not occur 
during the experiment. The participant was directed to "read the article as you would normally 
do." In accordance to the guidelines outlined in Ericsson and Simon (1993), the participant was 
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instructed to read and to "remember that it is very important to say everything that you are 
thinking while you are reading the text without explaining or interpreting your thoughts." During 
reading, the participant’s verbal reports were audio-recorded and another experimenter took 
detailed notes in regards to the participant’s non-verbal behaviours (e.g., turning pages, when 
sections were begun, his/her reactions to the text) in order to facilitate the transcription of the 
think aloud data. The participant made no markings on the article nor did he/she pause while 
thinking aloud for more than two minutes. The length of the session was approximately 45 
minutes. 

Coding and Scoring 

Wyatt and colleagues’ (1993) coding scheme was used to analyze the deployment of reading 
comprehension strategies that are specific to the social sciences while reading professional-level 
texts (see; Pressley, 2000; Pressley & Afflerbach, 1995; Pressley & Lundeberg, 2008; Wyatt et 
al., 1993). The coding scheme outlined in Wyatt et al. (1993) is based on a grounded (i.e., 
inductive) approach to studying the cognitive processes used by highly skilled readers in the 
social sciences. In order to capture all of the processes that occur while reading, the researchers 
gathered data until no new cognitive processes could be identified (Pressley, 2000; Pressley & 
Lundeberg, 2008). Given that the participant in this study is a social sciences professor reading 
an empirical article, we believe that the codes obtained from this model are both internally and 
externally valid. The coding scheme includes the following nine categories: goal awareness 
(GA); awareness (AWA); planful (PLAN); monitoring (MON); relating information to prior 
knowledge base (RIPK); evaluative reactions (ER); going beyond the information given 
(elaborations) (ELA); integration (INT); and elucidation of discourse structure (EDS). The 
following five categories were excluded as part of the current study, due to the fact that they 
reflect behavioural and affective aspects of reading: linearity and nonlinearity of reading (LNR); 
written responses (WR); affective reactions (AR); nonverbal responses (NR); and other (OTH). 

We also extended certain aspects of this coding scheme on the basis of recommendations 
outlined by Azevedo and Witherspoon (2009) in terms of the operationalization of constructs by 
taking into account their granularity and valence. We also specified the valence of specific 
constructs (positive and negative) as well as identifying utterances where the participant was 
simply reading the contents of the article (coded as ‘READ’). The resulting coding scheme 
shows definitions and examples in relation to each construct. 

The data consisted of 39 minutes of audio recording of the participant’s concurrent think aloud 
while reading the article. The transcription yielded a corpus of 5,974 words. Segmentation was 
done based on the semantic features of an utterance as a means to determine segment boundaries 
(see Chi, 1997). In total, the segmentation yielded 90 instances of reading, 94 utterances and 
wherein 167 segments were identified, each of which were assigned a code from the formalism 
shown in Appendix A. 

Coding Reliability 

To establish reliability of the think-aloud coding scheme, a randomly selected sample consisting 
of 20% of the total utterances was coded independently by two raters. Cohen’s kappa was 
calculated on this sample to assess inter-rater agreement, and was determined to be good (k = 

.68, SE = .08). Disagreements were resolved through discussion. Following this, the remaining 
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utterances were divided between the two raters to code independently (i.e., 40% of the total 
amount of utterances each). Frequencies of each code for all utterances were then tallied. 

Results 

Transient Analysis of Reading Comprehension Strategies 

In order to examine the temporal deployment of macro-level reading comprehension strategies, 
Table 1 shows the total frequencies of strategies used by the expert (i.e., rows) over eight 5- 
minute segments of time (i.e., columns) that make up the total amount of time spent reading the 
text (i.e., 39min: 14 sec). Given that the expert read the article from beginning to end, the position 
of each 5-minute segment roughly corresponds to the linear position in the text (e.g., the eighth 
and last segment represents the end of the journal article). The most frequently used strategy by 
the expert was simply reading a segment of text, which was found to be used more frequently all 
throughout reading except during the last 5-minute segment 1 (i.e., overall, reading accounted for 
90 occurrences out of a total of 257). During the last 5-minute segment, the expert engaged more 
frequently in evaluative reactions. Content evaluation was also the second most frequently used 
strategy overall (49 occurrences out of a total of 257) and was most predominantly employed 
during the second, third, seventh and eighth 5-minute segment. Monitoring and relating 
infonnation to prior knowledge seemed to co-occur especially during the first, second, fifth and 
seventh 5-minute segment (i.e., 27 and 32 out of a total of 257, respectively). Elaborations, 
affective reactions, and planful were used less frequently (15, 15, and 10 out of a total of 257, 
respectively) while integration, awareness, goal awareness, elucidation of discourse structure and 
other strategies were rarely used (less than 5 out of a total of 257). Therefore, the nature of 
highly-skilled reading comprehension is characterized by evaluating aspects of the text and its 
relevance to the reading goal, evaluating one's own understanding of the text, as well as 
activating prior knowledge in order to make sense of the text. 

State-Transitions in Reading Comprehension Strategies 

Likelihood metrics have been used in previous research to capture the temporal deployment of 
learning strategies during learning (see Azevedo, Moos, Johnson, & Chauncey, 2010; D’Mello, 
Olney, & Person, 2010; D’Mello, Taylor, & Graesser, 2007; Witherspoon, Azevedo, & D’Mello, 
2008). This form of state-transition analysis is perfonned by creating a matrix of all the possible 
learning strategies in order to calculate the probability estimate of transitioning from one state to 
another. 

In order to derive a likelihood metric that appropriately captures the temporal deployment of 
reading comprehension strategies, we opted to distinguish between the frequency and probability 
of shifting from one strategy to another (e.g., frequency and probability of generating an 
evaluating reaction following reading a segment of text). Frequencies in shifting literacy 
strategies was calculated by counting the number of times a literacy strategy "X" at Time / lead 
to another strategy "Y" at Time i+l. Probabilities in shifting literacy strategies were calculated 
by dividing the frequencies by the total number of time shifts in strategies originated from 
literacy strategy "X". In doing so, we were able to calculate the percentage of variability in 


The last interval corresponds to 4 minutes and 14 seconds as opposed to the other 5 minutes interval used for the 
rest of the table. Judging by the length and frequency of reading comprehension strategy used during the last phase 
of reading, we judged that this gap would have little impact on our findings. 
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strategy use that is accounted for by a particular type of shift (see Figure 1). Therefore, 
frequencies reflect the overall importance rather than probability. 


Table 1. Total raw frequency of reading comprehension strategies characterizing highly 
skilled reading used over time 







Time 




Comprehension Process 

T1 

T2 

T3 

T4 

T5 

T6 

T7 

T8 

Overall 

Reading 

13 

13 

12 

11 

8 

10 

17 

6 

90 

Elucidation of discourse structure 

0 

0 

0 

0 

0 

1 

0 

0 

1 

Integration 

0 

1 

2 

1 

0 

0 

1 

0 

5 

Awareness 

0 

1 

0 

0 

0 

3 

1 

0 

5 

Elaborations 

1 

3 

2 

2 

2 

1 

0 

4 

15 

Affective reactions 

0 

3 

1 

0 

3 

2 

4 

2 

15 

Planful 

1 

3 

1 

2 

3 

0 

0 

0 

10 

Monitoring 

4 

6 

3 

3 

5 

2 

4 

0 

27 

Evaluative reactions 

5 

9 

8 

4 

2 

5 

9 

7 

49 

Relating information to prior 
knowledge 

7 

6 

2 

1 

5 

2 

6 

3 

32 

Goal awareness 

2 

0 

0 

1 

0 

0 

0 

0 

3 

Other 

0 

2 

0 

0 

1 

1 

1 

0 

5 

Total 

33 

47 

31 

25 

29 

27 

43 

22 

257 

Mean (SD) 

2.8 

3.9 

2.6 

2.1 

2.4 

2.3 

3.6 

1.8 

21.4 

(4.0) 

(3.9) 

(3.7) 

(3.1) 

(2.5) 

(2.8) 

(5.1) 

(2.6) 

(25.9) 


Note. Times are reported in five minute intervals with last interval stopping at 39min and Msec; 
Linearity and Nonlinearity or Reading (LNR) excluded from the analysis. 
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Figure 1. Reading comprehension strategies state-transition matrix: Computing 
probabilities in single shifts in strategies 


Single-State Transitions. 

In regards to the transitions involving reading, the results of the single-state transition analysis 
show that highly skilled reading is characterized by shifting from reading to evaluating aspects of 
the text (24 occurrences accounting for a third of all strategies used after reading). In addition to 
evaluative reactions, which were found to lead more often to reading, engaging in monitoring or 
evaluating one's own understanding followed reading segments of the text more often, which 
suggests that it might be closely tied to aspects of the text (19 occurrences accounting for a fifth 
of all strategies used after reading). Relating information to prior knowledge occurred equally 
often before and after reading (12 occurrences accounting for a tenth of all strategies used after 
reading). 

In regards to the transitions involving strategies, the results of the single-state transition analysis 
show that there is a great amount of variability in the strategies used following evaluative 
reactions, since elaborations occurred most frequently but still only accounted for a small 
percentage of variance (4 occurrences accounting for 8% of all strategies used after evaluative 
reactions). The transitions originating from monitoring, however, were more homogeneous with 
evaluating one's own understanding of the text and critiquing aspects of the study occurring most 
often (each of them occurring on 4 occasions and accounting for 15% of all strategies used after 
monitoring). Finally, relating information to prior knowledge was found to be frequently 
followed by evaluating aspects of the study (8 occurrences accounting for 25% of all strategies 
used after relating information to prior knowledge). 
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Table 2. Frequencies and probabilities in one shift in reading comprehension strategies 
characterizing highly skilled reading 


Comprehension Process 

Frequency 

(Probability) 

Evaluative Reactions —> Reading 

32 (65%) 

Reading —> Evaluative Reactions 

24 (27%) 

Reading —> Monitoring 

19(21%) 

Monitoring —> Reading 

13 (48%) 

Relating Information To Prior Knowledge —> Reading 

12 (38%) 

Reading —> Relating Information To Prior Knowledge 

12(13%) 

Reading —> Affective Reaction 

12(13%) 

Relating Information To Prior Knowledge —> Evaluative Reactions 

8 (25%) 

Elaboration —> Reading 

8 (57%) 

Relating Information To Prior Knowledge —> Relating Information To 
Prior Knowledge 

7 (22%) 

Affective Reaction —> Reading 

7 (47%) 

Reading —> Elaboration 

6 (7%) 

Reading —> Awareness 

4 (4%) 

Evaluative Reactions —> Elaboration 

4 (8%) 

Monitoring —> Monitoring 

4(15%) 

Monitoring —> Evaluative Reactions 

4(15%) 

Evaluative Reactions —> Relating Information To Prior Knowledge 

4 (8%) 

Evaluative Reactions —> Evaluative Reactions 

4 (8%) 


Note. Top 10% ranked frequencies (18 of 144 shifts); Frequency = Count of shifts from Process 
x at Time i to Process y at Time /+1; Probability = Count of shifts from Process x at Time / to 
variable v at Time i+\ divided by the total count of shifts related to Process x (2> . Linearity and 
Nonlinearity of Reading excluded from the analysis. 


2 The data provided in the table consists of only a sample of all the data included in the analysis. 
This sample corresponds to 18 out of 144 data shifts. The sample was selected on the basis that 
the highest frequencies are shown. These frequencies correspond to the count of shifts from state 
“X” to “Y”. For example, the expert shifted from reading to elaborating on 6 occasions. The 
probabilities consist of the frequencies divided by the sum of shifts from “X” (i.e., to Yl, Y2, ... 
Yn). For example, the shift from Reading to Elaborating corresponds to 7% of all shifts from 
Reading. 
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Table 3. Frequencies and probabilities in two shifts in reading comprehension strategies 
characterizing highly skilled reading 


Comprehension Process 

Frequency 

(Probability) 

Reading —> Evaluative Reactions —> Reading 

15 (17%) 

Reading —> Monitoring —> Reading 

11 (12%) 

Evaluative Reactions —> Reading—>■ Evaluative Reactions 

11 (23%) 

Evaluative Reactions —> Reading —> Monitoring 

8(17%) 

Relating Information To Prior Knowledge —> Evaluative Reactions —> 
Reading 

7 (22%) 

Reading —> Relating Information To Prior Knowledge —> Reading 

6 (7%) 

Reading —> Affective Reactions —> Reading 

6 (7%) 

Evaluative Reactions —> Reading—» Relating Information To Prior 
Knowledge 

5 (10%) 

Reading —> Elaborations —> Reading 

4 (4%) 

Evaluative Reactions —> Reading—> Affective Reactions 

4 (8%) 

Elaboration —> Reading —> Evaluative Reactions 

4 (29%) 

Relating Information To Prior Knowledge —> Relating Information To Prior 
Knowledge —> Evaluative Reactions 

4(13%) 

Evaluative Reactions —> Relating Information To Prior Knowledge —> 
Relating Information To Prior Knowledge 

3 (6%) 

Reading —> Relating Information To Prior Knowledge —> Relating 
Information To Prior Knowledge 

3 (3%) 

Reading —> Affective Reactions—> Relating Information To Prior 

Knowledge 

3 (3%) 

Relating Information To Prior Knowledge —> Reading—>■ Relating 
Information To Prior Knowledge 

3 (9%) 

Affective Reactions —» Reading—> Evaluative Reactions 

3 (20%) 

Reading —» Monitoring—>• Monitoring 

3 (3%) 

Monitoring —> Reading-^- Monitoring 

3(11%) 

Reading —> Other—> Reading 

3 (3%) 

Elaboration —> Evaluative Reactions-^- Reading 

3 (21%) 

Monitoring —> Evaluative Reactions—> Reading 

3 (11%) 
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Note. Top 1% ranked frequencies (22 of 1728 shifts); Frequency = Count of shifts from Process 
x at Time i to Process y at Time i +1 to Process z at Time i+2; Probability = Count of shifts from 
Process x at Time i to Process y at Time i+l and Process z at Time i+2 divided by the total count 
of shifts in regards to Process x. Linearity and Nonlinearity of Reading excluded from the 
analysis. 


Two-State Transitions. 

The examination of the two-state transition analysis triggered by reading a segment of text 
corroborates the findings obtained in the analysis focusing on a single shift. In descending order, 
there is a high likelihood of shifting from reading to: evaluative reactions (i.e., ranked 1st overall 
with 15 occurrences), monitoring (i.e., ranked 2nd overall with 11 occurrences), and relating 
information to prior knowledge (i.e., ranked 6th overall with 6 occurrences); and then shifting 
back to reading (i.e., accounting for 17%, 12%, and 7% of all two-state transitions triggered by 
reading, respectively). 

Furthermore, the two-state transition analysis triggered by the use of strategies shows evidence in 
favour of the critical role of reacting through evaluating aspects of the study in highly-skilled 
reading. There was a high likelihood of using either evaluative reactions (ranked 3rd overall; 11 
occurrences accounting for 23% of all two-state transitions triggered by evaluative reactions) or 
monitoring (ranked 4th overall; 8 occurrences accounting for 17% of all two-state transitions 
triggered by evaluative reactions) following reading segments of text triggered by evaluating 
aspects of the text. Finally, using prior knowledge was also instrumental in evaluating aspects of 
the text followed by reading (ranked 5th overall; 7 occurrences accounting for 22% of all two- 
state transitions triggered by using prior knowledge). 

Qualitatively Depicting the Reading Comprehension Strategies that Mediate 
Expertise 

In order to provide converging evidence for the state-transition analysis and further describe the 
characteristics of highly skilled reading, we validated this quantitative analysis by substantiating 
it with a two-pass approach to coding the qualitative data (Chi, 1997). First, we identified the 
expert’s reading goal, which was to “I want to find out what this group is doing in particular [...] 
and one of the reasons for reading this, I want to know what they are doing especially since we 
are getting into eye tracking and we’re using that in our lab.” Second, the protocol was searched 
for occurrences of the desired state transition of literacy strategies (i.e., Reading —> Evaluative 
Reactions —> Reading) as a means to describe their properties at a finer grain level as well as 
their valence and aspects related to their content. 

Table 4 shows that the micro-level processes of Evaluative Reactions to the text. More 
specifically, reactions were focused on aspects of the research methodology that was reported, 
the writing and editing of the paper, the analyses, conclusions, and the citations used. Only one 
of the eight evaluative reactions had a positive valence meaning that the expert judged this aspect 
of the analysis as being a strength (e.g., "Oh that’s good, ok"). The others had a negative valence 
meaning that the expert had identified a limitation in regards to a certain aspect of the study, 
which was judged to be problematic (e.g., "This doesn’t sound like a very well controlled 
experiment"). 
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Table 4. Qualitative data coded during the second-pass for Evaluative Reactions 




Sequence of Literacy Strategies 

Valence 

Focus 

1st - Reading 

2nd - Evaluative 
Reactions 

3rd - Reading 

l (-) 

Writing/ 
editing style 

Tasks with a complex, 
dynamic visual component 
require not only the 
acquisition of conceptual/ 
procedural but also of 
perc eptual/attentional 
skills. 

I’m wondering why they 
put conceptual and 
procedural with a slash. 
They’re not used 
interchangeably but ... 

This study examined 
expertise differences in 
perceiving and interpreting 
complex, dynamic visual 
stimuli on a performance 
and on a process level, 
including perceptual and 
conceptual strategies. 

2 (-) 

Methods 

Performance, eye 
movement, and verbal 
report data were obtained 
from seven experts and 14 
novices. 

Ok, so I see that we’re 
heading towards a problem 
with unequal sample size 
um... 

Results show that experts 
compared to novices attend 
more to relevant aspects of 
the stimuli. 

3 (-) 

Writing/ 
editing style 

In many domains, expert 
performance also comprises 
perceptual/attentional 
skills, that is, the ability to 
perceive the relevant out of 
irrelevant information. 

Hm ... well, ok, 
information is used in a 
weird sense there, euuh... 

[...] highly visual stimuli 
and to draw inferences 
based upon the perceived 
information. 

4 (-) 

Methods 

This was done in order to 
avoid an artificial situation 
for both groups. On the one 
hand, novices might not be 
able to describe the 
locomotion pattern after all 
after a too short 
presentation. On the other 
hand, experts were forced 
to look at a stimulus that 
they had interpreted 
already. 

Geez this doesn’t sound 
like a very well controlled 
experiment anyways, ok. 

Eye tracking Tobii 1750 ok 
50 Hz ClearView software 
the verbal data were 
recorded by Camtasia 

5 (-) 

Method 

At the beginning, the eye 
tracking system was 
adjusted to the individual 
features. Before watching 
the videos while watching 
the video, please take a 
look at the way the fish 
swims. Subsequent to 
watching the video, you 
will have to describe the 
fish’s locomotion pattern. 
You will also be allowed to 
watch the video as often as 
you like. 

Ok so there is- there’s no- 
there is a goal but there is 
no explicit goal, so in terms 
of methodology I’m 
wondering, I mean, well 
you know euh it could be 
more precise but that’s just 
being critical of the 
method. 

Then, participants watched 
the looped video while their 
eye movements were 
recorded until they stopped 
it themselves. After having 
watched the video, 
participants were asked to 
describe the locomotion 
pattern of the depicted fish 
verba lly [... ] 


Deriving Empirically-Based Design Guidelines for Advanced Learning Technologies 


12 













CJLT/RCAT Vol. 38(1) 


6 (+) 

Analyses 

We refer to these 
aggregated AOIs as 

AAOIs. 

Oh that’s good, ok. 

The first dependent variable 
was the mean viewing 
duration per video. 

7 (-) 

Conclusions 

In addition, findings 
obtained from this study 
were intended to inform the 
instructional design of 
process- ok ah - hypothesis 

1 experts would perform 
more accurately and faster 
than novices on a 
locomotion description task 
- the respective test, 
however, mostly served as 
a manipulation check. 

Wow yeah, I mean that 
hypothesis is you didn’t 
have to do this study to do 
that but let’s move right 
along. 

More important, based on 
prior findings with static 
complex visual stimuli. 

8 (-) 

Citations 

More important, based on 
prior findings with static 
complex visual stimuli. 

Ok why is this stuff so old? 
Why are they citing old 
research? 

[...] hypothesized that also 
in this dynamic domain the 
process data of experts 
would show that they 
attend more to relevant 
information than novices, 
who would rather attend to 
perceptually salient, but 
potentially conceptually 
irrelevant. 


Therefore, reacting to aspects of the study after and before reading was characterized as a highly 
critical form of strategy. As such, the second-pass through the concurrent think aloud protocol 
enables us to validate our initial analysis that informed the creation of visual depictions of highly 
skilled reading. In order to foster expertise, we created visual representations of the reading 
comprehension strategies that mediate highly skilled reading as a means to provide students with 
double-content examples to facilitate skill acquisition. Each segments of the concurrent thi nk 
aloud protocol can be shown as boxes showing the contents of the highly skilled reader's 
utterances (see Figure 2). Each box is interrelated according to the order in which its content was 
uttered. Moreover, it is indexed in regards to whether the expert was reading or verbalizing the 
use of a particular strategy and its location in relation to the text. As such, these visual depictions 
can represent the state transitions in the use of reading comprehension strategies as a means to 
foster expertise in reading professional-level texts. These visual depictions of highly skilled 
reading can also be embedded in the context of a computer-based learning environment (Figure 
3). In doing so, learners can be assisted through technology-based scaffolding mechanisms in 
studying examples of the key reading comprehension strategies that were found to mediate 
highly-skilled reading. 
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Figure 2. Visually depicting shifts in reading comprehension strategies that mediate highly- 
skilled reading 


Discussion and System Development 

The objective of this study was to examine highly skilled reading in the social sciences as a 
means to derive theoretically-based and empirically-based design guidelines for advanced 
learning technologies. Specifically, we expected that the literacy strategies used by social 
scientists reading while reading professional-level texts could be visually represented as worked- 
examples of heuristic solution strategies. Our findings indicate that highly-skilled reading in the 
social sciences is characterized by critiquing text information, relating infonnation to prior 
knowledge, and evaluating one’s own understanding of text information. In the following 
sections, we review the theoretical and empirical literature on example-based skill acquisition. 
We then discuss the implications for the design of the Highly-Skilled Reading Tutor, a single¬ 
agent intelligent system that assists students in studying double-content examples of the reading 
comprehension strategies that are the most characteristic of highly-skilled reading in the social 
sciences. 

Using Example-Based Skill Acquisition as an Instructional Model 

In order to support undergraduate students in learning about the reading comprehension 
strategies that mediate expertise in the social sciences, we use example-based skill acquisition as 
an instructional model (see Atkinson, Derry, Renkl, & Wortham, 2000; Renkl, 2005, 2010; 

Renkl & Atkinson, 2003, 2007, 2010). The main assumption of the instructional model states 
that the combination of worked examples with principle-based self-explanation prompts 
facilitates skill acquisition. Based on cognitive load theory (Chandler & Sweller, 1991; Sweller, 
1994; Sweller, Van Merrienboer, & Paas, 1998), the worked examples are expected to increase 
the amount of cognitive resources available in working memory. In contrast to the worked 
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examples, principle-based self-explanation prompts ensure that additional cognitive resources are 
directed towards reflecting on the rationale that underlies the skill illustrated by the example. 

Renkl, Hilbert, and Schworm (2009), and van Gog and Rummel (2010) have recently begun to 
use worked examples to demonstrate different types of heuristic strategies such as deriving 
mathematical proofs and fonnulating arguments. Reading professional-level texts in the social 
sciences consists of an ill-structured task, since it involves the acquisition of complex knowledge 
through the application of domain-specific concepts and skills. Examples of the skills used in 
this domain distinguish between two types of elements: The first, which is the learning domain, 
consists of the problem-solving structure (e.g., the strategy used to read the text, such as 
critiquing an aspect of the methodology); while the second, which is the exemplifying domain, 
refers to the problem-solving topic (e.g., the topic of the text being read, such as reading about 
the methodology used in the study). 

Skill acquisition can also be facilitated by providing novices with principle-based self¬ 
explanation prompts while they are studying the examples (Schworm & Renkl, 2006, 2007; 
Rummel et ah, 2006). Principle-based self-explanation prompts assist students in generating 
explanations in relation to the use of heuristic strategies, thereby relating the novices’ prior 
knowledge to the learning domain that is depicted in the worked example. For instance, self¬ 
explanation prompts that focus on the learning domain can prompt students by asking them 
“Which reading strategy does this example illustrate?” as well as “How is it related to the 
reading goal?” 

Salden, Koedinger, Renkl, Aleven, and McLaren (2010) used computer-based learning 
environments as a means to implement and evaluate the effectiveness of interventions while 
investigating the impact of example-based skill acquisition on skill acquisition. The existing 
empirical research that example-based skill acquisition is an efficient and effective means to 
facilitate the acquisition of skills involved in well-structured domains such as geometry and 
chemistry (McLaren, Lim, & Koedinger, 2008; Salden, Koedinger, Renkl, Aleven, & McLaren, 
2010; Schwonke et al., 2009). Indeed, computer-based learning environments offer numerous 
advantages, such as controlling the sequence of examples based on students’ performance in 
generating self-explanations as well as providing corrective feedback. 

However, the effects of example-based skill acquisition in the social sciences, as opposed to 
other ill-structured domains, have yet to be investigated (Hilbert, Renkl, Schwonn, Kessler, & 
Reiss, 2008; Hilbert, Renkl, Kessler, & Reiss, 2008; Schworm & Renkl, 2006). Therefore, the 
design of computer-based learning environments based on example-based skill acquisition as an 
instructional model has a significant potential to facilitate the acquisition of the reading 
comprehension strategies that mediate expertise in the social sciences, thereby contributing to 
both scientific knowledge and practice. 

The Highly-Skilled Reading Tutor 

The Highly-Skilled Reading Tutor is a computer-based learning environment that is designed to 
facilitate the acquisition of the reading comprehension strategies that mediate expertise in the 
social sciences. The elements of design for the Highly-Skilled Reading Tutor include the 
pedagogical agent module, the text module, and the example module. 
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b J Highly-Skilled Reading Tutor (Hi-Sla Tutor) 
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Many instructional materials directly or indirectly convey 
expert knowledge to learners tFeldon. 3007). For instance, 
a very effective instructional technique is the use of wirked 
examples, in which novices are shown a wurked-oul expert 
solution procedure (Swelter. Van Mcmenboer. & Paas. 1998). 
However, worked examples have so far mainly been used to 
convey conceptual and pocedurnl aspects of expertise. In 
many domains, expert performance also comprises perceptual/ 
attention!I skills, that is. the ability to perceive the relevant out 
of irrelevant information in complex, highly visual stimuli and 
to draw inferences based upon the perceived information (e.g., 
X-rays in medical diagnosis see Lesguld ct al. 1988: weather 
maps in meteorology see Can ham & Hcgarty. 30101 

An important question is, if these perceptual/attentionaI 
aspects of expertise, that is top-down processing of perceptual 
stimuli, could also be conveyed to novices to facilitate skill 
acquLsi tion. Since providing infirmatim al a conceptual level 
has been shown to be effective for the acquisition of 
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conccplual/procedural drills, it might be necessary to j 
novices mure direct access to the perceptual/al ten ti 
processes underlying experts' perfotma 
guiding the novices' attention to critical perceptual info 
tion during the study of wurked examples based on evid 
regarding experts' perceptual processes, such as eye m 
merits l Van Gog. Jorodaka. Scheiter. Get]civ & Paas. 200*1 
process-oriented worked examples see Van Gog. Paas. & 
Merrienbocr. 3008). However, as will be discussed in the 
section, not much is known about how experts allocate 
attention during task performance, and how their alter 
allocation differs from novices, especially in domains 
involve complex, dynamic visual stimuli. Therefore, 
preseil study examined expertise in the perceptual/al tend 
and conceptual processes involved in the inleiprelalio 
complex dynamic visual stimuli in the domain of 
locomotion. 

1.1 Expertise differences in perceiving and interpreting 
complex visual stimuli 

Many studies have addressed the issue of how ex 
perceive and interpret visual stimuli by using eye true 


Many instructional materials directly or indirectly convey 
expert knowledge to learners. For instance, a very effective 
Instructional technique Is the use of worked examples, In 
which novices are shown worked-out examples. However, 
worked examples have so far mainly been used to convey 
conceptual and procedural aspects In expertise. 



In many domains, expert performance also comprises 
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Figure 3. Screen capture of the Highly-Skilled Reading Tutor 


The pedagogical agent, also called artificial agent, can interact with students to facilitate 
knowledge construction through the use of verbal communication (Graesser, Jeon, & Dufty, 
2008). Pedagogical agents can simulate social interaction through different modalities such as 
speech, facial expression, eye movement, gesture, and posture (Dehn & Mulken, 2000; Johnson, 
Rickel, & Lester, 2000; Kim & Baylor, 2006). In a review of the literature, Graesser, D’Mello, 
and Person (2009) indicated that pedagogical agents can parallel and extend the adaptiveness and 
effectiveness of one-on-one tutoring by demonstrating the appropriate use of learning strategies. 


The text module enables students to load any social science manuscript in an Adobe Acrobat 
Reader embedded as part of the environment. Students are thus able to read the text while having 
the benefit of the features of this software, which include the ability to focus or zoom-in or -out, 
review the structure of the paper, and view annotated comments. 


The example module provides students with double-content examples of reading comprehension 
strategies (i.e., visual representation of highly-skilled reading in the social sciences, also saved in 
a .pdf format and shown in an Adobe Acrobat Reader interface). The example includes reference 
to the part of the text the expert was reading in order to enable students to examine the reading 
path of the expert. 

Based on example-based skill acquisition as an instructional model (Renkl, Hilbert, & Schworm, 
2009; van Gog & Rummel, 2010) and cognitive load theory (Chandler & Sweller, 1991; Sweller, 
1994; Sweller, Van Merrienboer, & Paas, 1998), the instructional principles that guide the 
pedagogical agent aim to assist students in studying worked-examples of highly skilled reading. 
The Highly-Skilled Reading Tutor directs learners’ cognitive resources to activities that are 
relevant to learning (i.e., maximizing gennane load through principle-based self-explanation 
prompts provided by the pedagogical agent) rather than to processing irrelevant information (i.e., 
reduce extraneous load through studying worked out examples of highly skilled reading). By 
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analogy to the recent advances in tutored problem-solving with intelligent systems in well- 
structured and algorithmic domains (see McLaren et al., 2008; Salden et ah, 2010; Schwonke et 
ah, 2009), the principle-based self-explanation prompts are combined with corrective feedback 
and a glossary as a means to focus learners' attention on the learning-domain or literacy 
strategies, thereby avoiding to induce extraneous cognitive load. Furthermore, the worked 
examples of highly skilled reading are adaptively faded based on the learners’ correctness of 
responses generated to the principle-based self-explanation prompts. 

The Highly-Skilled Reading Tutor stands to better prepare social scientists who are capable of 
highly-skilled reading, a skill which is critical in planning, conducting, and reporting leading- 
edge research. Future research will examine the nature of highly-skilled reading in the social 
sciences by manipulating aspects in relation to the text and reader as a means to (1) further 
develop the worked examples embedded in the Highly-Skilled Reading Tutor and (2) implement 
and evaluate the effectiveness of the Highly-Skilled Reading Tutor in terms of enhancing 
learning. 
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